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FOREWORD 



The Bureau of School Programs Evaluation of the New York State 
Education Department has for a number of years been developing proce- 
f^^^^^^^^^f^i^g the performance of schools and school districts. 
The Quality Measurement Project, which began in the late 1950's 
assessed the academic .achievement of school districts in relation to ( 
'ill til ^•^""^^^^ili'^y' parents' education, and fathers' occupations^ 
The Performance Indicators in Education project, starting a decade later, 
used other socioeconomic factors in the community to develop expected 
achievement levels for school districts against which the districts- 
actual achievement levels could be compared. Thus, • a 'district 's per- 
formance was based, not on how the district compared with other dis- 
cricts, but on how it compared with a unique standard derived from its 
own characteristics. 

Paralleling this interest in the output of school districts has 
been a concern £ot the processes operating within school systems. To 

■ iXr^''''^f 'y'^ °" ' ^^^i^" '"^de of almost 

100 studies dealmg with factors related- to student performance. Two 
reports -resulted: one, a detailed description of methods and results-^ 
the other, a summary which related research findings to ten important' 
questions about the effectiveness of education. 2 

With this /background, the bureau launched its own studies of ' 
school processes to learn more about what schools can do to improve the 
ach.evernent of students. Three ba^ic strategie's were used. One strategy 
involved carrying out statistical analyses of data available in the 
Department s data files to determine how various school factors relate 
to student achievement. Both the second and third strategies involved 
stra pf^'"^ f:. . 1°"-P"f"™ing schools. Then, under the second 
strategy, available data for the schools were analyzed to find variables 
which distinguished the twcv .roups. The third strategy called Yd 
observing in high- and 1 ,er.Wng schools to discover classroom 
processes which differentiated between the contrasting groups of schools. • 

The present report is a summary of these studies. The studied 
have a common purpose: to identify -relationships between school factors 
and student performance. While the reader is urged to be cautious in 
inferring cause and effect from these relationships, it is hoped that 
the information resulting from the studies will be useful in setting 
policy and making decisions. j ' \^ 

ThP n. J"^'' document is a result of efforts of a number of individuals. 
The Outlier Study, described in Part III, was carried out by Austin D 
Swanson, Professor of Educational Administration, and Robert C Nichols 
Professor of Educational Psychology, both of the University of New York' at 
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Buffalo. The Observational Study, described in Part IV, was carried 
out by Richard M. Clark, Professor of Educational Psychology of the - 
University of New York at Albany, The Regression , Studies were carri,ed 
out 'by the staff of the Bureau of School Programs Evaluation: David 
J. Irvine, Chief; Gerald H. Wohlferd; Guy D. Spath; and Philip J. 
Pillsworth. Geqnaro DiGiovanni, who served as a public administration 
intern during the time these studies were under way, conducted^ several 
of the regression analyses: Mr. Spath coordinated the consolidation 
of *the several studies iVito this report^^^^^ ^ 

1 . ' • 



John W. PoUey , 
Associate Coiranissioner for 

Research, Planning, and - 

Evaluation - 
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Do Expenditures Affect Learning? 

Several types of expenditures were investigated. Since the 
largest single expenditure for education is for salaries, teachers' 
Salaries were studied in several different ways. 

The Regression Studies revealed a positive relationship between, 
median teacher ^5^1aries of school districts and average student achieve-' 
ment in the disLr^icts. These findings were supported by the Outliers 
Study, in which teachers in schools achieving aWe expected earned the 
most money and teachers in 'schools achieving be low expected received 
the least. While these findings do qpt mean that higher salaries ^ 
produce hij^her performance, they indicate that the more succec^sful dis- ' 
tricts do pay higher salaries. Not unexpectedly, salary-related variables, 
such as graduate credits and experience of teachers, were also 
shown to be related to achievement. It 'is 'interesting to note that an 
earlier study, using 1969 data, showed that the amount 9f monej^ spent , 
per pupil on principals' salaries was also rela ted* to -Achievement. 

The Regression Studies also examined full tax value and several 
district per-pupil expenditures, including total expenditures and expen- 
ditures for regular day instruction, for teacher^, for central adminis- 
t:ration\and for principals. No consistent, relationships were found, 
in spile of. the f(»ct that wide variafions were observed from district 
Lo di sli ict . 



How Arc S r^cj_,)J^J^) o^^Vc^ms and 

Service^ Related Co^ Achiev emonr.* 

Special programs frequontly showed negative relationships with 
achicvonienL. The Outlier Study, for example, revealed - negative rela- 
tionship between achievement and special programs for n.. handicapped,. 
Guidance, social work services, and attendance services were also nega- 
tively related to achievement, while programs for the academically 
talented were more ^likely to be found in the high outlier school's. These 
findings can be misleading if caupe and effect are assumed. Sudi programs 
are most likely to be found where they are most needed. Therefore, it 
is not surprising to find that njorc' special programs are found where 
achievement is low. In additioYi, the data available for th^se studies 
did not clearly distinguish between types of special programs; it is 
likely that some types of programs affect students diff-erently from 
other types of programs. • * • 

In the Observational Study, high-achieving schools were rated as 
having higher total activity in nine of eleven reading activities, 
• especially 4n silent reading, than did Ibw-achieving schools. * The Out- 
lier Study showed that the use of rooms for academic rather than voca- 
tional study. ,\lso correlated positively with achievement. The Regression 
Studies indicated a positive relation between attendance rate and achieve- 
ment, but this relationship .disappeared when socioeconomic factors \>fcxe • 
considered. The same phenomenon occurred in a study of student mobility. 
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PART I 

AN OVERVIEW 0^ THREE SETS OF STUDIES 
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PARX I . • , 

AN OVEFWIL'W OF TlllU-K SETS OF STUDIES ' 

f ^ 

Three sets of studic-s of .school factors were conck.cled tTncler the ' 
aegis of the Hurenu of School ProRrams Evaluation. All of the studies 
had the, purpose of idontifyini; school factors which relate to M.o achi^ive- . 
ment of students. Pcirt I of Ihis report desciVibcs the three sets of ' " 
I. studies and summarize r _the findi^igs.' 

I ' ' "S • 

_ This report wa^ designed to complement an eavl ier' bureau publi- 
/ cation.. UKLt_Re.Si^ch Says About_Jnin:ov:Lng S-tudcnt Pci fc^ Vmance . 2 That 
publjcnt lon-'sumiuu-ized the rd-sults of almost 100 researriTluijie s ' ■' ' 
deal in;; with the ^relation of a Variety of school factors to scjiool aut- 
comes. .The present report attempts to integrate results of research 
done more recently in the scliools of New York State. ' > ■ 
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fn. \ \r T\ ro^ult^ reported here can provide- a basis 

for thoughtful discussion and, together p;i th other informalibn , suggest 
direct roiTS which can be pursued to imprtSve education. By its ve'r'y 
•nature, the research does not offer simple answers to the complex ques- 
tions confronti-ng education today But if the findings, seem to b6 ' 
cautiously inter proted-, perhaps tfk^t fact will help. avoid rushing to 
conclusiorts or jumping on bandwagons. ' 



THE STUDIES ' . ' ■- 

•> . " . ' 

Regres s ion Studi os 

f'"' f''.°^ studies, carried Out by the bureau staff, inves- 
tigatod factors which are at least partLy under the control of school 
personnel. ^ As m the Coleman Report, 'the effects' of nonschool factJbrs ' 
were controlled for and the unique contributions of school variables • 
were studied. The studies as a. group are refeirred to as the Regressi on 
bLudicvs and are described in more detail in Part II. — . 

T he Outlier - Study " • ' ' . v " • 

The second typo, of study used multiple regression analysis to < 
Identify schools which were performing either above or below their pre- 
dicted levels of achievement, as computed from nonschool factors." The' 
study focused on school buildings rather than scjoool districts. Three 
groups of schpols were idenfified: 1) High o^JfTHers, those .schools 
Whose actual mean achievement scrfres were.Wi?ll above their predicted 
scores' 2) Low outliers, those schools whose actual scores weVe well 
below their predicted scores; and 3) MidUers, those schools whose . * 
actual scores fell near their predicted scores. Those three groups of 
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schoolwS were compared through analysis o£ variance iii an attem^ji: to 
determine'^the effects of a number of variables which reflect school 
processes,* Because ,this^ study emphasized schools v7hich lay some dis- 
tance, from' their expected levels of achiovoment , it was dubbed Cho 
Outlier Stu dy. A more complete def;cription can bo lound ' in Part TIJ-. 

The Observational Stu^ ly ^ 

— 1 f/ • . . . J 

The third kinyof stucty involved observing in lA sciiools whicl) 
were identified as above or below predicted achievement in the Outlier 
Study described above, Obs^srvatipnal instruments, interviews, and 
questionnaires. were used to obtain informatign about classroom activ- 
ities, interactions apiong studcn,ts and staff- members, and percopLions 
of staff members.- This study is referred to as the Observational St;udy 
and is described more fully in Part IV. 



^ FINDINGS 

While caeh of the sets of studies described nboxc i>s^-l n vonc:- 
what different approach,* their findings are couBolldnted \n.]o\: in v.n 
attempt'^to develop a coherent, though not necessarily complete, picture 
of how school processes relate to school outcomes. Where the results 
axdrfOiiUddiclcr^ or ambiguous, ^ an attempt is made? to shov this, - 



t ^Is Si^e a Kactor? * < ^ 

The. average -chool district enrollment in No'w York Srat^ in 1971 
and 1972 was approximately 2500, rangim^ from 30 to ov< r 30,000, In 
the Regression Studies, district enrplbiteat was found to be ne.vttiveiy 
rel/atcd to achievement; that is, larger districtii had poorer average 
achievement. However, when total population of districts considered, 
it replaced enrollment as a predictor, Thid seems to suggest that the 
hegative relationship between enrollment and achievement is a function • 
bf 'urbannw.s v ahor than of school size, Thc-finding, thou, docs not 
jappear to offt c evidence about the optimum' si-/-e of schools or school 
jdistricts. * 

t> 

''lie Ouillor Study showed no difference in size between high and 
low outlii T sc.aols; each gi'oup avera^^ed about 100 fewer students than 
did schools idcotllied as midlicrs. In additioiyf^lasses tendec to be 
smaller for .both positiv^ tnd negative outlicrs\than for midliois. As 
'has been suggested, in oLh^-r studies,, the effect of class sixe may be 
dependent on" the typo, of student and on tlie subject being taught, ^ 
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Do Expenditures Affect Learning? " . * . 

^ * * 1 

Several types of expenditures were investigated. Since the 
largest single expenditure for education is for salaries, teachers' 
Salaries were studied . in -several different w,ays . 

The Regression Studies revealed a positive relationship between... 
median teacher -^^larics of school districts and average student ac^iicvb-' 
ment in the districts. These, findings were supporL^ed by the Outliers 
Study, jn which teachers in schools achieving above expected cfarned the 
-most money and teachers in-schools achieving below expected received 
'the least. While these findings do qpt mean that higher .salaries ^ . 
produce higher pei^'formance, they indicate that the more succec^sful dis- * 
tricts do pay higher salaries. Not unexpectedly, salary-related variables, 
such as graduate credits and experience of togchers, were also 
shown to be related to achievement. It*is Interesting" to note that an 
earner study, using 1969 data, showed that the amount 9f mone^ spent , 
per pupil on principals' salaries was also related* to Achievement. 

The Regression Studies also examined full tax value and several 
district per-pupil expenditures, including total expenditures and expen- 
ditures for regular day instruction, for teacher^, for central adminis- 
trationV^and for principals. No consistent, relationships were found, 
in spite of the f^ct that wide vnriafions were observed from district 
to disLi i.ct . ^ " , 



Ilovj Are Special Pr otj;rgms_gnvi 

Services Related to" Achiev argont-J^ 

Special programs frequently sliowed negative relationships with 
achievement. The Outlier Study, for example, reveled ^ negative rela- 
tionship hcLween achievement and special programs for u... handicapped.. 
Guidance, social work services, and attendance services were also nega- 
tively related to achievertent, while jjrograms for the academically 
talented were more likely to be found in the high outlier school-s. These 
findings cah.be misleading if caupe and effect are assumed. -Sudi programs 
are most likely to be found where they are most needed. Therefore, it 
is not surprising to find that njorc." special programs are found where 
•achievement is low. In additioYi, the data ay.ailable for th^se studies 
did not clearly distinguish between types of special, programs; ^it is 
likely that some types of programs affect students diff^erently from 
other 'types of programs. 

In the Observational Study, high-achieving schools were rated as 
having higher total activity in nine of eleven reading activities, 
especially -in silent reading, than did low-achieving schools. * The Out- 
lier Study, showed that the use of rooms for academic rather than voca- 
tional study .j^lso correlated positively with achievement. The Regression 
Studios indicated a positive relation between atcendaftce rate and achieve- 
ment,, but this relationship. disappeared when socioeconomic factors were • 
con^sidered. The same phenomenon occurred in a study of student mobility.' 
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Tfraditional vs. Opeif Classrooms 

Attempts to move away from ''traditional , teacher-centered class- 
room arrangements have included many innov'ations designed to increase 
learning. The Outlier Study showed that most classrooms in the schools 
studied were traditional, with the smallest number of traditional 
classrooms being found in low outlier schools and the greatest number 
in midlier schools. .High outliers were 'more likely to have open class- 
rooms and multi-age groupings. Multi-unit plans were negatively re- 
lated to perf ordnance. Midlier schools were less likely to have in- 
nbvative programs than either of the other two gi;oups of schools. 
Other organizationaf arrangements were not -significantly related to ♦ 
achievement. These included clustering, continuous progress, depart- 
mentalization, differentiated staffing, dual progress plan, hoiise plan, 
modular scheduling, non-graded, self -^contained , and team teaching. 
The Observational 'Study supported the findings of the Outlier Study # 
in respect to open classrooms. 

* Do Teache)- Characteristics Make' a Difference? 
\ j" 

,i ,Vhen VantKLdates for teaching positions walk into a superintendent's 
office, they bring with ^them certain personal and professional character- 
istics. 'Among them ar^ their professional training and experience., age, 
sex, marital status,^ arid, a variety of personal traits.' The superinten- ' 
dent may have ver^ little concrete evidence to use in selecting amoilg 
candidates with an almost infinite number of combinations of professional 
and personal characteristics. Does a candidate with a doctorate have 
more to* offer than one with many years of experience but Tess formal 
education? 'Are women more effective than men in elemeritary scKoojeS? 
How important is graduate school t|:aining vhen the teacher is expected 
to teach reading to eight-yi^ar-old children? 

The. New York State Education Department collects data on teachers ^ 
in eacfh of the school districts in the ^ate. From^the available data, 
five teacher characteristic variables were seLected for study: 



■4. 



Median Age of Teachers in .the District 
Median 



2. Median Years of Experience 

3. Percent of Married Teachers ^-j 

4. Percent of Male Teachers 

5. Percent of Teachers Having^Gr^duate Credit 

The Regression Studies revealed no relationships between teacher 
age or experience and average^: student achievement in a district." The' 
Outliers Study, on the other hand, showed a positive relationship 
between teacher experience and student .performance. Furthermore, a 
larger percentage of the teachers in the' high outlier^ schools were 'on 
tenuVe. 

■15 ■ • ■ 
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The percent of teachers having some graduate training showed thb 
most consistc'ht rolntionship wit.h Reading and<maf.hemotics achievcinont 
\n the Regression Studies. Kven allter controlling for socioeconomic 
factors, ^percent of teachers having 30 or more, credit hours beyond the 
bachelor s degree demonstrated a sigi>ificant relationship to all of the 
achievement criteria. 

SigniCican't shifts in teacher certification took place from 1970 
r.o 1971. In 1970 only >230 districts had all teachers certified while 
m 1971 this number increased to over 480. The" Regression Studies of 
this variable showed a definite negative relationship of percent uncerti- 
fied to student achieveno.nt in 1970. The tremendous reduction in number' 
of schools having uncertified teachers in 1071 led to less conclusive 
resjilts for. that year,, but the inference still ieems warranted that 
certif ic'aCion is desirable. ^ 

The positive findings on certification and graduat^e training were 
both supported in the school-building analyses carried out in the Outliers 
Study . 

V 

Using regression analysis to study school buildings, it was found' 
that schools with a greater percentage of Black teachers had higher levels 
of performance, after controlling for non-school factors. ' 

The findings on sex and i^iari tsl status wore mixei. In" 1971, no 
relaticxnship v/as found between percent 'of married toacheVs in a district ' 
and achievement. In 1972, a positive relationship was"' evident . . Percent 
of male teachers was negafively related, to all achievement criteria in 
1972 -but only to 'sixth-grade mathematics in 1971. 

V 

While these results suggest that relationships exist be- 
tween student achievement and teacher graduate training, it should be 
notes! that these relationships may reflect other 'factors related to both 
achievement and teacher characteristics. For instance, we know that 
low- socioeconomic sc'-ool districts tend to have low loan scores on achieve- 
ment tests and high socioeconomic districts tend to h^ve high mean scores. 
The positive relationship that seems to exist between percent of eeachei's 
with graduate credits and student achievement may simply reflect a ten- 
dency among higher socioeconomic districts to employ teachers with gradu- 
ate training. Conversely, the lower socioeconomic districts may not 
have the money to pay the higher sa^laries of teachers with graduate 
credits. , ' 

The hy^)othesis that these teacher characteristic relationships 
are merely reflective of the known relationship between socioeconontic ' 
status and achievement is even more tenable regarding teacher marital' 
status and sex. Lower socioeconomic districts might be expected to 
hire more men since women might be^unwiUing to teach in those districts. 
And, with a higher teacher turnover rate, these districts could be ex- 
pected to have a lov;er percentage of married teachers. ~ > 
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How Do School Conditions 

and Practices Influence Learnin?^? 

Logic suggests that school conditions and practices which impinge 
directly on the daily lives of students offet the most promise for 
improving education. Yet in many ways they are the hardest to study. ^ 

The Observational Study attempted- to obtain data which would 
make it possible to jjnderstand better the importance of certain con- 
ditions and practices. Some of the results arc summai-izod below: 

1. Teachers in- high .outlier schools made less overt effort to 
maintain class control', had less rigid student behavior, but were more 
efficient i.t\ maintaining the ^ level of control they appeared to want 
than were teachers in low outlier schools. 

2. Teachers in high outlier schools were rated as warmer, more 
responsive, and placing more emphasis on cognitive development. 

/ ^ 

3. ^ MoreC^total activity takes place in reading classes in the 
high outlier schools. 

1, 

4. Children In high outlier schools engage in more silent 
reading while children in low outlier schools engag<^ in more oral 
reading. 

5* In grades one to three, teachers in high outlier schools 
gave more positive and less negative reinforcement than did teachers 
in the Tow outlier schools. 

6. In gradys four to six,, detachers in low outlier schools gave 
more reinforcem^m. In general, however, they tended to use, negative 
reinforcetncnt n)ore* than the teachers in the high outlier schools. 

7. Pupils in the high outlier schools were more' enthusiastic 
and were better *able to sustain attention. 

8. On selected items, related to open education, the high- out- 
lier schools appeared more often. 

9. Items on physical space and facilities generally did not 
differentiate between high and low outlier schools. 

Hoy Do the Attitudes of ^ ' * \ 

the Staff Relate to Learning? i 

The attitudes of teachers are frequently cited as influences on 
the performance of students. For that reason a number of attitudes, 
perceptions, and expectations of teachers and other staff members were 
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investigated in the Observational Study. The findings include the ' • 
following: 

' 1. Teacher's in high outlier schools expected more children to 
graduate from high school, to go to college, to become good readers ' 
and to bcco?!ie good citizen^ than did teachers in low outlier SQ,hools-, 

2. Teachers in high performing schools saw their children as 
more intelligent, better behaved, more pleasant to teach, and their 
parents as more concerned. ' . 

3. Teachers in high and low outlier schools were not different 
in the amounts of help they perceived as being available in handling 
problems, 

4. Reading teachers in the high outlier schools gave more £avor- 
able evaluations of the reading programs in their scUools than dld*^ 
reading teachers in low oatlier schools. They also rated the class- 
room teachers more favorably in using' appropriate materials, extending 
reading into ofJier areas, asking children to read with purpose, and 
using informal diagnosis. . 

5. Principals in high outlier schools generally ScW their ' 
personnel as more competent, than did principals in low outlier schobls. 

• 6. Principals' in high outlier schools saw themselves as havjing 

better rapport with teachers, pjirents, and pupils than did the priiici- 
pals in low outlier schools. However, prihcipals in low outlier sfchools 
reported better rapport with the school bpard. ' 

* ' • » 

' . SUMMARY OF FINDINGS •! 

* * ' I i 

Summarizing the findings of a group of related studies presents 
a number of difficulties. The volumq^of -results resists a concise 
treatincnt. Attempting to discuss, the results in siinple terms may pro- 
duce .misleading conclusions. The ambiguities and contradictions between 
the findings of different studies niay defy easy explanations. 

A particular- problem iq interpreting cross-sectional data, which 
those studies us.id, involves the extent to which variables can be 
infx^rrQd to cause the outcomes with which they are associated. It is 

- P''''^.^^ ^^•'^ ^^^^"^y °^ Research that "correlation does not.lmpiy causation. 

_;;iH9.w.ever, there is a human tendency to jumi) to conclusions about cause 
and. effect. Findings such as those presented Here should be interpreted 
with restraint and logic in order to avoid faulty conclusions. 

One area in which a logical analysis of the situiition may avoid 
incorrect conplAisioas has to do/^ith the findings that 'special programs 
are frequently associated witj/low achievement. 'A hasty conclusion may' 
be that the special prograws/are ineffective or actually detrimenfal to 
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student achievement, ftowever, many special programs--f or the handi- 
capped or for the diswivantaged , to cite two instances--have been 
implemepted to meet /articular needs. A negative correlation between 
th6 prevalence of special programs and achievement, rather tlian 
meaning that the programs have adversely affected achievement, indi- 
cates that the programs arc located where tliey hve needed. * 

> • 
In spite of the px)ssiblc problems-of interpretation , it seems 
that a sumniary of 'findings from thq three types of studies i\fay be 
useful. The followinj^ suniaiary shows which variables were associated 
witfl) students* achievement in reading and arithmetic after social and 
economic factor's were accounted for. 



Factors Assaciat e d with Hi^h Achieveme nt 

High achicvemqnt^ of students was associated with the following 



factors: 



1. Higher teachers' salaries. ' 

2. Use of rooms for academic riither than vocational 

study. • 
3. . Open, classrooins. 

5. Multi-age grouping^. ^ 

5. Higher levclc of graduate 'training of teachers. 

6. Larger percent of Black teachers. 

7. IjGtter control of classes but , with less o<7ert 

effort on the part pf teachers to maintain 
control . 

8. L^3ss rigid student ^behavior. >. 

9. Greater teacher -warmth and responsiveness. , 

10. Greater^ emp}ui5>is by teachers on cognitive ^ 

development . 

11. More total activity in reading classes. 

12. More silent -reading. . ' ^ 

13. ' Positive reinfc^rcement students byi teachers. 

14. More enthusiasm on the part of students. 

15. Better ability on the part of students 

>to sustain attention. , ' * 

16. Higher expectations on the part of ^ teachers 

for their students to become good readers 
and good citizens, to graduate from high 
• school, and to go to college. 

17. Teachers* perceptions of their students as more 

intelligent, better behaved, and more pleasant 
to, to^ach and the students* parents as more 
concerned. 

18. More favorable ratings by reading teachers of the 
^ reading program in their schools. 
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19^ More fayorable ratings by reading teachers of 
classroom teachers in using appropri,ato 
materials, extending reading into other ar'^.as, 
asking children to j;iiii4^7ith- purpose, and 
using inCormal diagnosis. 

20* Higlier ratings by principals of the competence 
Qf personnel* in their schools. 

2]. Principals* {jerceptions of a high level of 

rapport v7iLh teachers, parents, and. pupils. 

B^ic tors .,Ag^ocijijW 

" ** — - *■ 

f 

Low achievement of students was associated with the following 
factors: ' 

,ar^er district enrollment. 
Special programs for the handicapp.^»d. 
Pupil services, including guidance, social 
services, and atte,ndance sfervicos. 
A. Mnlti-upit groupings. , 

5. Larger percent of uncertified teachers In a 
district or- school. 

6. More oral reading. 

7. Negative reinforcement of students by teachers. 

8. Principals* perceptions of' a high level of 
rapport with the school board. 




jVicJror£^j3Jio\d^^ Rolationsbipo wit h Ach lovr>m^>h^ 

A number of variables Showed ambiguous relationships with 
student achiev^mc»nt . Among these were: 

1. ClassXslKG. 

^ 2. Pcr-puijil expenditures for instruction, teachers, 

pVincipals, and central .administration as ^11 
as total per-pMpil expenditures. ^ 
^3. Median years of experience of teachers in a' district. 
•4. "Percent ^of married to'achers in a district. ^ 
5. Percent of male teachers in a district.' 

Factors Not Associated with Achievement * 

Several variables were found not to be associated with achieve- 
ment. 'Among these v;ere: » ^ . * 

^* Attendance rate, once socioocjnrtomic factors are o 
I considered. ' 

2. Student mobility. 
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3. ^Various organizational "*and grouping arrangements, 

including clustering, continuous progress, departy^.* 
mentalization, differentiated ^staffing, dual progVe^s 
pl^in, bouse plan, modular scheduling, non-^graded, ^ 
^elf-contained, and' team teaching. 

4. Median age of teachers in a district. 

5. School facilities and. space. 

6. 'Teachers' ^perceptions of the amount of help available 

in handling problems. 



/ ' DISCUSSION ' ' T\ 



In many respects, thft findings of these studies seem to agree 
with other research conducted during the past, decade. However, some 
very logical, l^ng-held assumptions about *the effects of certain system- 
wide administrative variables are luot supported. None of the '^expected" 
relationships with achievement were demojistrated for attendance, 
mobility, special con\2ensatory programs * and services, and gross expen- 
diture variables,, - ; ^ 

Mixed results jv/ere foUnd for class siz^e, school siz^e, and inno- 
vative programs. Smaller classes in smaller schools and innovative 
programs appear to be ^ found in both high and loi^ outliers. 

The most encouraging, findings related to feacher ^aracteristics, 
staff attitudes, and^schodl practices. Results of the several studies 
indicate that good teachers are the heart of the educational system, as« 
conventional'wisdom would suggest. Students seem more, likely to achieve 
.well where teachers are better strained, more often certified, higher^ 
paid, and more likely to be tenured. The teachers in high performing 
schools have higher expectations for their students and more favorable., 
perceptions of them; they appear to be war^ner, more supportive, and more 
responsive: This is accomp'anie.d by more enthusiastic Stu9ents. Teachers 
in move successful schools als9 .apjpear to deemphasi^ze strict control 
and lean toward more open educfition. These studies do not inform us as 
to what "better" teachers do to bring about high achievement, nor do 
they explain why a warmer, more open ''environment is more conducive to 
learning. 

The relationships described are not necessarily causal. ^General- 
izations drawn from the-se studies should be considered in the light of 
other research and the decision maker's experience and unique situation. 
These .findings are offered as one more^bit of information which can con- 
tribute to an understanding of educational processes. One conclusion 
seems c^pparent: Studying district-wide variables and school var.iables 
seems to be less rewarding than studying the teaching-learning interface^ 
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THE REGRESSION STUDIES 



The Performance Indicators in Education (PIE) , project was initiated 
by the State Education Department with the purpose of developing new ways 
of studying the performance of schoqi^s and school disti^icts. 'The major 
thrust has' been toward reporting school -^dlstrict performance while taking 
into account the characteristics of the dist>rict and its 'students. 

School district performance was estimated by analyzing data col- 
lected from various' sources including the U. S,' Census, the Basic Educa- 
tional Data system, and the Pupil Evaluation Program (PEP). The primary 
statistical tool used in. the project is multiple regression, analysis. 
By analyzing socioeconomic d^ta and achievement data ^r school distrio^ts, 
combinations of variables were idejitified which predict: average district 
scores on reading and mathematics , as measured by thfi^EP tesj^. Each 
set of relationships was expressed as an equation, ^wiiiih included a weight 
for each of the variables contributing to the prediction of .achievement • ^ 
'By substitiiting in the equation the values pi the predictor variables for 
a given district, a predicted score waa^obtained for that district on that 
pjarticular measure of achievement. The differertce between the ^c'tual 
score for the district and this predicted score was its performance indi*» 
cartor. 

Performance indicators provide a better estimate of school dis- 
trict performance than does the mean achievement £est score for the x 
district, since they represent factors outside the control of the schools 
which affect student performance,. We know, for instance, that socio- 
economic status correlates highly with school achievement. This suggests 
that a school which has a heavy concentration of student's from high socio- 
economic families is likely to have some educational advantages not shared 
by all districts. To compare such a district's test .scores with tl^ose of 
districts with different characteristics is not very meaningful. 

To summarize, the origipal intent of the Perfbirmance Indicators 
Project-was to take into 'account each district's unique characteristips 
so that^ a more realistic picture of the district's effectiveness coul^ 
be obtained. A more detailed descriptioh of the procedure ^an be found 
in the technical manual prepared £or the 1974 report.^ , V ' 



Studying School District Processes 

\ • » 

The models developed to assess district performance ^escribe 
the status of a district's functioning in certain areas of its curric- 
ulum. In ordet to improye the performance of school districts,, it is 
necessary to determine What causes the district to function as ^t does 
and to infer what chang^es will bring about improvement/* 

t 
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The performance Indicators models served as a foundation for 
studying school processes. They Were used to coatrol statistically' 
for the effects of nonschool factors in order, to study the r.ela't ion- 
ships of school conditions ,and resources to pupil athievement.' • ' 
. ^ ^ . ^ ' ] ' ' ^ 

Variables used to control for npnschool factors included: 

- T\;^tal Tor-tlation of the District ' • * 

Percent Rural* Population * , * 

Percent of Children Living in> a * ^ ^ ^ 

Fa:her/M6ther Family " ' ^ ^- 

t Percent of Owner Occupied Incusing Units 

Percent of Population Living in Units ^ t 

with 1.01 PejTson or Hove Per Room 
Percent of Population Living in Units 

Lacking Spme Plumbing / ^ 

State Aid Ratio 
Prior Achievement 

The actual equations, their parameters, scatter plots, and graphs are 
available in the technical manual referred to above. 

^ ■ ' . ' / 

While the PIE equations were developed using a stepwise techni- 
^que, the process variables were studied using the Full and Restricted 
Models approach as described by Bottenberg 'and Ward*. The Full 
Model for a given criterion includes the socioeconomic variables of 
the original equation plus the process variable of interest. The 
Restricted Model is the original^ equation without the process variable* 
The percentsof variance accounted for on the criterion by the Full and 
Restricted Models are compared. The difference is the Unique portion 
of variance which can be attributed to the process variable. An F test 
can- be applied to test the statistical significance of this unique > 
contribution as follows: 

' .F = (Ri - 4)f^Pl . 

(1 - Ri)/df2 ^ ' • 



where: Rj^ the squared multiple correlation of the Full Model 
('i.e., the percent of variance on the criterion ^ 
accounted for by all the variables). 

R2 = the squared multiple* correlation of th^ Restricted ' 
Model (i.e.., the percent of variance on the "criterion 
accounted for by the original PIE equatipn). 

• . dfj^ = the number of linearly independent variables in the 
Full Model less the number in the Restricted Model. 

q.f2 =. the number of cases or observations less the number 
of -linearly independent variables in the Full Model. 
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The simple correlation of a process variable to each control 
variable must also be considered due to the problem of colinearity. 

The methodology described above is most conservative* The unique 
contribution alone is tested. Any conmionality of process variables with 
socioeconomic factors is attributed to the control variables • The 
Coleman ReportS has received .extensive criticism for using a stepwise 
approach that does not takp into effect the colinearity between back- 
ground and school process, t'igure 1, representing real data on teacher 
certification, demonstrate^' the problem. In 1971 the unique cony^^ 
bution of a teacher certification variable to the criterion was ira^tis- 
tically significant; however, the magnitude of the contribution * 
quite small. When one visualizes the statistical relationships in 
graphic form, as suggested by Mayeslce et al >^ in a reanalysis of the 
Colilpan.data, it is apparent that the first-order correlation (repre- 
sented in Figure 1 by unique plus shared variance) of percent uncertified 
teachers with performance is of a much greater magnitude than the unique 
contribution alone. This cannot be noted in the regression technique 
described, since most of this variance is conmion to or shared with the 
background variables. 

r , 

Figure 1 

1970 'Percent Uncertified Teachers and 
r971 Third-Grade Mathematics With 
Socioeconomic' Factors Controlled 
* • 
. Total Variance on Third-*Grade Mathematics 




Unaccounted for variance 

Variance uniquely accounted for by socioeconomic factors 
Variance uniquely accounted for by percent uncertified teachers 



== Common or shared variance 
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The method described above was applied to fifteen* school district 
variables. Results were reported in several Bureau papers. 7,8,9, 10,11,12,13,14 
The major findings are described below. 

■ ' ' ! 

Attendance Rate ^ • * ^ 

A significant 'positive relationship was found between district 
attendance rate and achievement; however, when socioeconomic factors were 
controlled, tl\e relationship disappeared. This suggests that the rela-' 
tionship is primarily a. result of socioeconomic differences between dis- 
tricts. ' ^ ^ 

•A ■ * ' 

Mobility ^ • 

District rate of student mobility showed a significant negative ^' 
correlation with achievement in a study of QO districts. The .fact that 
the relationship disappeared when socioeconomic factors were considered 
indicates that,* in districts of similar socioeconomic makeup, mobility is 
unrelated to achievement. * ^ 



Enrollment 
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En>-ollment was negatively correlated with achievement, but, when^ 
district population was entered as a control variable, enrollment made 
no unique contribution. This suggests that enrollment is an indicator 
^of community type rather thah a factor that can be manipulated^ to 
improve achievement. - : 



Teacher Characteristics 

No relationship was found between median age of teachers in a^ 
district and student achievement as measured* by mean district Pp? scores. 
The same is^true for median years of teacher experience; Analyses of the 
relationships of percent of married teachers and percent of uncertified 
teachers resulted in contradictory findings in different years. 

A negative relationship was found between. percent of male teachers 
in a district and 16 of 28' PEP tests and subtests.. 

Finally, the positive relationship between achievement and percent 
of teachers with graduate training remained after accounting for socio- - 
economic factors. * 

Care should be used in interpreting these results. A school super-* 
intendent considering hiring a teacher with 20 years of successful teaching 
experience .should not automatically reject the applicant because^median * 
experience do^ not correlate "with achievement. The results are more likely 
to be useful in developing policies regarding the hiring of "teachers and 
encouraging graduate training than in making decisions about individual 
teachers. ^ ^ 

.26 ' ^, ■ ' ■ 



Similarly, the fact that low achieving schools tend to have a 
higher percentage of male teachers e^ven when socioeconomic status is 
controlled for does not imply that male teachers in any cause low 
achievement. It does suggest that the nature of that relationship 
"^should be more thoroughly examined by educational researchers. 

Finally, the strong relationship between percent of teachers 
with graduate credits and achievement tends to confirm the conven- 
tional wisdom regarding teacher training.* Op the-basis of these 
findings,, it is not unreasonable to suggest that districts continue 
providing itjcentives for teachers to take graduate courses. 



Teacher Salaries \ * . ^ . 

On all reading and mathematics tests and subtests, me4ian 
teachet:s^salary showed a unique relationship beyond that accounted 
for by socioeconomic variables, suggesting that salaries are not 
merely a reflection of a distiict's ability to pay. Figure 2 
illustrates two of ^ these relationships. Principalis' salaries also 
appear to be related to performance. 



* Figure 2 

Relationship of Median District Teacher Salary 
To Reading and Mathematics Achievement 
With Socioeconomic Factors ^Controlled 

Total Variance on 1971 PEP 



Third-Grade Math 



\ 

4» 



In 





Sixth-Grade Reading 



I Unaccounted for variance 
Variance uniquely accounted for by socioeconomic factors 
Q Variance uniquely accounted for by median teacher salary 
S Common or shared variance 



27 



•19- 



♦ I 

Conclusion 



, The. findings suiranarued above indicate the inexactness of our 
knowledge of school processes* Additional results of the regression 
studies, are not reported here because of their ambiguity. Even 
fairly straightforward and consistent results should be interpreted 
cautiously for several reasonfe: 

1. The findings show relationships between variables but ^not 

what causes what. 

• ^ ^ ^ # 

2. The findings are based on one set of school districts (in 
most cases, all the districts in New York State) at one or more times 
in tihe past. The same results may not hold for other schocl districts 
or for other time periods. 

3.. The findings show relationships which have been fouqd for 
the school districts as a group, but the* relationships may not hold 
true for any individual district. 

Nevertheless, the findings provide a starting point for con- 
sidering changes which are intended to improve the performance of 
pupils in two areas of the curriculum. The findings can be used to 
supplement knowledge obtained from other sources. For example, a 
school superintendent who is trying to decide between two candidates 
for a position may w^t to consider the findings on graduate training 
and experience, but also he would certainly use his knowledge of other 
qualities needed in the position .in making his decisiorv. 

It is hoped also that the^ findings may serve as a stajrting point 
for additional researqh. Research is needed to analyze more thoroughly 
school processes., to obtain more satisfactory indicators of student 
performance, and to establish causality between school processes and 
student performance'. 
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PART III 

f ■ 
THE OUTLIER STU£»V 

' The research reported in Part H used the school district as the 
,unit of study. This part of the report describes the strategy used 
and th<2 results obtained in al study of school buildings. 

• 

A Strategy for Studying School Processes 

One ySt^Jproach to the study of school processes is to compare 
"good" schools with "poor" schools on a number of school' factors. 
But identifying good and poor schools has been a problems 

School quality is sometimes defined in terms of the performance 
of the students in a school* However, research by a number of investi- 
gators has indicated the extent to which^ student^* performance is 
related to factors over which the-school has ^little or no control. 
Perhaps the most well known of this research is the Coleman Report. ^ 
TherefiOre, examining the performance' of students without considering 
those/ factors not controlled by the schools is.lfkely to produce 
inaccurate information about the role the' school played in determining 
students* performance. 

The major thrust of the PIE project has been to attenpt to evalu- 
ate districts after controlling for differences in these influential 
community factors. It seemed reasonable, then, that "good" and "poor" 
schools might jbe identified by the degree to which they exceed or fail 
to attain the level of achievement which community factors indicate is 
reasonable , to expect. 

The strategy decided upon to test this hypothesis included the 
following steps: 

1. Compute expected levels of achievement for schools in the . 
state, using socioeconomic variables and other factors not controlled 
by the schools. 

2. Identify those schools with actual average achievement scores 
which vary most from tbeir expected achievement, both positively, (high- 
performing schools) and negatively (low-performing schools). These 
schools have been dubbed "outliers." 

3. Compare high-performing schools with low-performing schools 
on a number of variables which describe school processes • 
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* Determining Performance of Schools 

A contract) was negotiated with two faculty members of the State 
University of New York at Buffa-J-o, through the Faculty of Educational 
Studies^ to identify schools with exceptionally high and exceptionally 
low levels of performance and to investigate characteristics associated 
with performance. Their findings were included in two reports to the ' 
State Education Department. 

The first step In determining performance of schools was to 
develop criterion measures, A principal components factor analysis 
was made of school mean subtest scores for third- and sixth-grade 
reading and mathematics scores on the Pupil Evaluation Program (PEP) 
tests for the four school ye^rs 1969-70 through 1972-73.^ The analysis 
revealed a general f^actoj which accounted for 80 -percent of^'the total 
variance in school means on all subtests in all years* All subtests 
had correlations with the general factor greater than .80. This 
indicated that no large amount 6l variance among school means on ariy 
test in any year would be lost by. retaining only one factor. 

Rather than develop an exact measure of the general factor^, an 
effort was made to develop a simple indicator which was clearly iden- 
tified with the underlying influences. This was accomplished by using 
the average of the reading and mathematics test scores for third and 
sixth grades as a general factor .score . Only PEP scores for 1972-73 
were used to avoid unnecessarily r^educing the number of schools that ' 
could be studied. This procedure produced criterion measures for 
2,624 schools. Included were public and private schools which/ con- 
tained both a^ third and a sixth grade. 

The strategy used to obtain, an e^cpected level of achievement on 
the criterion measure was to control for variables which could be con- 
sidered to be indicators of student background and were relatively 
immune to ^change as a result of the schools* efforts. A number of F" ch 
background variables were identified in the data routinely collected 
by the State Education Department through its Basic Educational Data 
System (BEDS). Among them were: 

1* Whether the school is public or private. 

2. The percent of filsjck students in the student body. 

3. The percent of Spanish-American students. 

4. The percent of students from families primarily 

supported by public assistance. 

X 

5. Geographical location of the school: a) New York 

City; b)* urban area other than New York City; 

c) suburb of a major city; or d) primarily rural. 
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Multiple regression analysis \9'^s used to determine how well • 
background variables predicted the criterion measure. Each of the 
variables listed above contributed sigiidf ic'antly to the prediction 
with the result that a regression equati^pn was formed which accounted 
for 69.6 percent ^of the variance of the criterion, measure. 

Using the equation, a predicted criterion score was calculated 
for each of the 2,624 schools wi*th both third- and sixth-grade PEP 
scores. The predicted criterion score was subtracted from the actual 
criterion score for each school to obtain a reoidual score. Thus, a 
high positive residual score* indicated that a school was achieving 
bel^ter ' than expecfed from the background variables. A high negat;ive 
score meant the opposite. Since the criterion measure was developed 
from rea(^ing and mathematics test scores for third and sixth grades', 
the residual was interpreted as a generalized measure of performance 
in the basic academic skills at the elementary school level. 

The difference between the actual and predicted scores was 
used to identify outlying schools. Schools which. had a residual score 
in excess, of +5 were classified as positive deviates. Those having^a 
residual greater than -5 were classified as negative deviates. Schools 
which had a residual ^core^ between .5 and -.S^wfrr-e classified as non- 
-deviates. The standard deviation of the residual ^sco^es was 3.161. 

The three groups of schools were subjected to^a one-way analysi3 
of variance oft 177 variables relating to pupil background, school loca- 
tion, professional personnel, program, and achievement. Parallel* 
analyses were made for private and public schools combined and for 
public schools alone. Personnel data were not available 'for private 
schools. 



Findings 

One hundred forty-eight schools were identified as positive 
deviates. Of these, 43 (29%) were private schools; 105 were public. 
In the negative deviate group, there were 145 schpols. Fif t;y' (347o) 
were private and '95 were public. In the non-deviate group, 104 (327*) 
were private and 219 were public. 

Table 1 reports the means and standard deviations for the three 
groups on the criterion variable and on the control variables (the 
independent variables in the equation predicting achievement). As 
would be expected, there "is little variation among the three groups 
within eacfi analysis 6n the Control variables. The total group 
averages about 127o Black enrollment vith approximately 1/6 of the 
schools exceeding 30%. Spanish American enrollment averages over 6% 
with approximately 1/.6 exceeding 20%. The public schools average more 
than l27o of their pupils on welfare; the percentage is lower for 
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private schools. Priv^ate schools ^so tend to enroll a smaller per- 
centage of studfents from minority groups. On the -achievement criterion, 
positive deviates average approximately one standard deviation above 
the total group mean; negativedeviates average approximately one 
standard deviation below.. The" average for non-deviates falls near the 
total group mean. Private- schools on the average achieve 'slightly 
higher than public schools. " • - 

Table 2 lists those school and program variables with a differ- 
ence among group means which is significant at a .05 level or above." 
Also reported are group means and levels o£ significance of differences 
between individual group means and the respective means of the other 
two groups combined. " ' ' • • - 

< 

Both positive and negative deviate schools average 'about 100 
students less than the non-deviates. Regardless of classification 
private schools, averaging 220 pupils, are much smaller than public 
schools, averaging 584 pupils. GeneraUy, as the analysis shifts from 
the total group of schools to public schools only, the direction of 
difference in school and program characteristics remains the same but 
in some instances the magnitude of thfej difference drops. It would 
appear. that private schools more than public schools tend to organize 
programs around the academically talented, multi-age grouping, and 
non»-graded classes. 

■ Positive deviates are more likely to have programs for the aca- 
demically talented, an open classroon^ arrangement, and multi-age 
grouping. They have' significantly fewer compensatory edocation programs- 
and guidance counselors. The number of students per classroom is likely 
to be lower for both groups of deviates. Traditional classrooms are 
likely to be found in most schools." However,' the proportion of traditional 
classrooms is lowest ir^ the negative devia'te schools. The non-deviate 
schools are generally less likely to have instituted what might be 
termed "innovative"" programs than either of the deviate groups. 

The differences in achievement statistics are consistent for the 
four years studied, 1969-70 to 197"2-73. Findings with reference to 
achievement are reported in Table 3 for 1971-72 only. For all years, 
all grades, and all subjects, achievement means are high for the posi- 
tive deviate schools and low for the negative deviate schools. The 
standard deviations are smaller for positive deviates than for negative 
deviates for all years and for all subjects except. sixth grade mathe- 
matics where the reverse is true for all years.- Most differences on 
the achievement means are significant at. the .001 level. 
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Data on the characteristics of teaching staffs are available 
only for public schools. The findings are reported, in Table 4. 
Positive deviate schools tend to have a smaller percentage of male 
teachers than do the other two groups (16%, Zff/. and 227. respectively). 
They also tend to have more highly trained staffs. Non-deviatBS have 
the fewest uncertified teachers, 3. SX, while negative deviates have 
the most, 5.67o. Positive deviates average 4.17<, uncertified teachers 
but they have the largest standard deviation, 10,7%, A larger percen- 
tage of teachers in positive deviate schools are on tenure than for 
the ot^er two groups (697»,65%and 59Z respectively). Teachers in 
positive deviate schools have more experience in the ^district and in 
total, years teaching. The means in average total experience for the 
threti groups ar*e 12', 11, and 10 years respectively. Teachers in 
positive deviate schools earn the highest salaries. Their counter- 
parts in the negative deviate schools earn the least. 



Conclusions 

The three groups of schools are tradit4onalJLy oriented; however, 
deviate schools, both pbsitive and negative, tend to be more innovative 
than non-deviate schools. Those who sti?.l. believe that schools can 
make a difference in children's learning can gather some comfort from 
the' analysis of teacher characteristics. 'Teachers in the positively . 
deviate schools as a group epitomize what conventional wisdom claims 
to be characteristic of good teaching staffs. They are better trained 
and more experienced. They are paid higher salaries|and are more 
likely to'^e on t^enure . This still does not inform usi as to what well 
trained, experienced and highly paid teachers do^ to bring about 
une^:pec tedly high pupil achievement. Such knowledge is essential if 
we are to prepare and organize teachers and the teaching processes to 
obtain optimum results. ' 
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PART IV.: 



THE OBSERVATIONAL STUDY 



An Ecological Study .of Teachers c^id 
Administrators in Fourteen Schools 
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PART IV 
THE OBSERVATIONAL STUDY 



Backja:round ^ ' • 

The Performance Indicators in Education (PIE) project demonstrated 
that all 'schools with low socioeconomic backgrounds do not do ^poorfy. 
^"H-a^^lysis of. 1970 Pupil Evaluation Program (PEP) reading an4 mathematics, 
scorefe by the Bureau of School Programs Evaluation revealed a ^ 
wide range of test scores among New York City schools/ even those drawing 
largely from low socioeconomic populations. ' 

An investigation wa5 initiated fo^ determine the extent to which 
school environment^ school program, ^nd administrative conditions might 
be related to test results. For the study, a group of five schools was 
identified in which reading scores were consistently- highland five . 
schools in which scores were consistently low. Also, two additional 
schools were identified for study.- In one, test scores had increased 
oyer a two-year period. In the other, scores had declined. \ ^ 

The twelve schools were then used as the targets* for structured 
observations, carried out under contract with Assessment Associates of 
Cortland, 'New York. On the basi^ of their observations, members df 
'evaluation teams classiffed eight out of ten of the high and low schools 
correctly. In seven out of ten cases, all' evaluators judged correctly. 
(Generally, three evaluators visited each school.) While 'specif ic> dif- 
ferences between high and low scores were not completely consistent, 
seven factors emerged as likely to be more true of high-achieying schools " 
than of low-achieving schools. Briefly, these characteristics were: 

1. Tea.chers manifested better rapport with students. 



2. 
3. 
4. 



Teachers exercised more effective control of pupils. 

Teachers engaged in more extensive preparation of lessons. 

Reading instruction was at a more appropriate level for 
the needs of pupils. 

Teachers provided for more extensive regrouping within 
• the reading period. 

Teachers provided for more extensive use of material in 
the. reading program. ' ^ 

The sources of leadership in instruction in reading were 
more forceful and ^positive. 
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A* second study was undertaken to: 

•1. Verify the findings of the earlier study. 

2". ^ Produce more refined instruments £or studying scJIIqoI 
processes . 

3. Expand the stady to inc^lude non-urban school districts. 

! 

^ 4. Eliminate several design problems of the earlier study. 

The study was carried out under the direction of a faculty member 
of the State University of New York at Albany. A description of the 
study and findings was included in a report to the State Education 
Department .^^ The major elements of that report are presented below. 

Instrument Development 

As a first step, the original forms used in the initial study 
were .examined. It was agreed that various items of demographic data 
such )as the socioeconomic level of pupils would not*Be stressed in the 
instruments to be developed. Rather, more attention would be given to 
aspects of teacher behavior and school environment that are under the 
control of the ^school. If any of the aspects noted were to be related 
in a meaningful way to differences m pupiL learning, stich aspects 
would be sc^ething that educators might do something about. From the 
original forms, a list was made of items* fq;:- further - cqrisideration. 
The importance of and the psychometric problems connected with each 
item were considered, as well as ways and means of gathering relevant 
information. Specific and general scales were constructed, edited, 
and used in field tryouts in two schools. In each case, four obseivers 
visited the school. Pairs of observers made twenty-minute o 'servations 
in six different classrooms so that each observer was paired with every 
'other observer. Also a pair of observers interviewed the reading teacher 
and the other two observers interviewed the school principal. After the 
field trials, data were inspected for reliability and other considerations, 
and the forms were revised. 

V 

Eight different instruments were developed. Four of the instruments 
were designed to be used in each of the classrooms visited . One observation 
instrument called for a general assessment of the school. Also developed 
were two interview schedules and a questionnaire to be filled^ out by the 
teachers observed. The names of^he instruments and a brief description' 
of each follow. 
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1. General Classroom Observation - This form was 
designed for rating the degree of existence of 
.various aspects of classroom emphasis, teacher 
behavior, pupil behavior, and facilities. It 
includes 16 five-point rating scales. 

•2, Teacher Questionnaire - This multiple choice 

f ortib was designed to elicit teachers' perceptions 
of -their students, the program, and adminis- 
trative support . 

3. Principal Interview - This instrument contains 

. a set of five-point rating scales for eliciting 
principals* opinions regarding .teacher effec- 
tiveness, adequacy of facilities, and de^^'^of 
their rap^o/ct with teachers, parents, students, 
and school board. 

4. Observation of a Reading Group - The form is a 
modification of a system developed by Educational 
Testing Service. ^0, 21 sixteen categories 
describing the reading prog:7am are scored on a 
Likert scale arranged from "little" to "much/* 

5. Reading Teacher Interview - The interview guide 
was adapted from an "Observer Guide— Reading, " 
published by the Bureau of Reading, New York 
State Education Department. Thirteen categories 
to be evaluated from "low" to "high" on a five- 
point scale were selected to describe the degree 
to which reading practices were seen as ideal. 

6. Teacher Reinforcement Scale - Likert scales from 
"low" to "high" were used to described the fre- 
quency and strength of both negative and positive 
reinforcement provided by the teacher, 

7. Characteristics of Open Education - This form is 

a shortened version of a scale developed by Walberg 
and Thomas22 to operationalize the definition of 
open classroom. It contains 1§ items using five- 
point Likert scales to measure the degree of 
openness. 
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8, General School Obs9;rvation - This form, using 
a Likert scale, calls for an overall assessment 
of morale, level of expectancy, and general 
* school appearance. 



Procedures for School Visits 

After instruments had been revised and constructs redefined, 
fourteen school buildings were selected on the basis of Xhe Outliers 
Study described in Part III of this paper. S^ven of the buildings 
were high outliers (positive deviates) and seven were low outliers 
(negative deviates). At this stage, the observers were unaware of 
whether the schools they visited were "high" or "low." 

For each school visit, the principal was asked to arr^ge access 

to nine elementary school classrooms between kindergarten and sixth 

grade. (In a few of the smaller schools, nine classrooms were not 

available.) Each observer was assigned to fonr of these classrooms, 

and tne pair of observers was together in one classroom during the 

day. Xn addition, interviews were held with the school principal and 

with a reading teacher In the school. The original design called for 

each observer to be paired with every other observer, but because of 

scheduling problems, this plan was not completely carried out. 
« 

Analysis of Data: Overview 

In the following. seven sections of this report, data are presented 
oji seven of the eight instruments developed for the study. For each 
instrument, mean scores are presented for (1)- first- to^ third- grade 
classes in the seven high-performing schools; (2) first- to third-grade 
classes in the low-performing schools; (3) fourth- to sixth-grade classes 
in the high-performing schools; and (4) fourth- -to sixth-grade classes 
in the low performing schools. Tests of significance are not reported 
for each pair of means, por are standard deviations provided. However, 
almost all data were obtained on five-point Likert scales. Most stan- ' 
dard deviations were very close to 1.00. In general, group sizes were 
similar. As a rule of thumb, differences between means of approxi- 
mately .50 points can be considered to be significant at the .05 level. 

In addition to mean scores, certain of the correlational rela- 
tionships are displayed and commented upon. With the great number of 
variables in the study, not all of the correlations are presented. 
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- Mann-I^itney U tests were uaed on selected segments of several 
devices in order to ascertain whether the devices significantly dis- 
criminated, on a rank order basis, between high and low schools. 

Filially, a root mean square procedure was used to test the 
extent of agreement between raters in this study. 



General Classroom Observations 



Data from the General Classroom Observation form reported in 
^Table 5 show clear differences between high-perf orming and low-per forming 
• schools in grades. 1-3 but relatively little dif fere ice in grades 4-6. 

For early elementary grades, items which did not differentiate 
betweq^n the .groups of schools were program emphasis on ^social develop- 
ment rigidity of student behavioi;^, and three items on 'facilities. 
High-performing schools were* signifipantly higher on all items except 
"effort to maintain control and "rigidity of student behavior." , For 
these ratings^ high-performing schools were significantly lower. Thus, 
in grades 1-3 classes in high-performing schools, teachers apparently 
. exerted less effort to maintain control and had less rigid student 

behavior, but at the same time yere rated significantly higher in effec- 
tiveness of control. Also teachers in high-performing schools Vere 
. rat^ed as warmer, more supportive, more responsive to students, and 

showing more- emphasis on cognitive development. Pupils in their classes 
Vappeared iftore erfthusia's tic about 'school and better able "to sustain 
attention. , 

In grades 4-6, however, only the differences in teaeher* efforts 
' to maintain control and effectiveness of coh^ol were significant. 

The pattern was the same as in the lower grad^T-^h teachers in high 
, reading schools making less effort to 'maintain control but being 

rated more effective in control. 
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Table 5 

Means for High- and Low-Performing Schools 
onjGeneral Classroom Observation Form > 



> 


Grades 1-3 ' 


Grades 4-6 


Kindergarten 


^ Item Number 


.High 


Low 


High 


Low 


High 


Low 


• 


Perfor- 


Perfor- 


Perfor- 


Perfor- 


Perfor- 


Perfor- 




ming 


ming 


ming 


ming 


ming 


ming 


Emphasis on: 












• 


1 .Cognitive Develop- \ 


2.9 


2.2 


2.9 


2.5 


2.8 


2.7 


ment 














2 . Language Deve lop- 


3.2 


2.5 


2.7 


2.7' 


2.8 


2.2. 


ment 














3 .Social Development 


2.3 


2.2 


2.1 


2.1 


3.0 


' 3.2 


• 

Teachei 'Behavior: 














4. Effort to Maintain 


1.7 


2.5 


1.5 


2.0 


1.8 


2.0 


Control 














5. Effectiveness of 


3.9 


3.3 


3.7 


3.3 


3 5 




. Control. 














6. Warmth of Pupil- . 


3.2 


2.7 


2.7 


2.9 


3.1 


4.2 


Teacher Interacjdm 














7. Amount of Inter- 














action with indi- 


'3.2 . 


2.7 


3.0 


2.6 


3.3 


3.3 


vidual pupils 














8. Amount of supf)orti;e 














or risk reducing 




9 9 


2 n 


9 1 




J.J 


activity 


2.9 












9. Responsiveness of 


2.1 


2.4 


2.3 


2.1 


3.0 


pupil * s ideas 














10. Use of pupil respcxi- 














ses to guide 


2.7 


1.9 


2.3 


2.3 


2.2 


2.5 


teacher's strategy 














Pupil Behavior 














11. Apparent enthusi- 


3.4 


2.9. 


2.7 


2.9 


3.6 


3.8 


asm for school 














12. Rigidity of pupil 


3.1 . 


3.8 


3.6 


3.4 


2.6 


3.7 


' beha^'ior 














13. Ability to sustain 


3.6 


2.8 


3.4 


.3.0 


3.3 


3.7 


attention 














Facilities 














14. Attractiveness of 


3.1 


2.8 


2.3 


2.6 


3.3 


3.3 


room displays 














15. Adequacy of space 


3.2 


3.4 


3.1 


3.1 


"3.8 


4.2 


16. Use of space 


3.1 


2.8 


2.8 


2.4 


3.4 


4.0 
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Teacher Questionnaire 

Teacher questionnaires were left in each of the schools visited; 
and a request was made to re'turn the questionnaires when completed to 
the Project Director. Forty-eight responses from high-performing schools 
and 51 responses ffom low-performing schools were available for anal;'sis. 
The first four items of the questionnaire concerned teachers* expectations 
for children in their classes. These data are reported in TaWe 6.' 



Table 6 

Mean Expectation for Pupils Expressed 
by Teachers in High- and Low-Performing Schools 
(Nine -point scale) 



Expectation for Pupil to 


High-Perfprming^ 
Schools N=48 


Low-Performing 
School^ N=51 


Graduate from high school 


8.85 


.' 7:36 


Go to college 


6.09 


' 2.73 


Become fluent reader ^ 


'7.17 


5.66 


Become a good adult citizen 


8 . 72 • 


7.89- 



As can be seen, teachers* in high-performing schools had l^ighex expectations 
for children in all four of the areas 'questionedr It is interesting to 
note in each groyp that teachers expected more children to graduate from 
high school than they expected to become fluent readers. <^ 

A similar pattern is seen in Table 7 which reports the results of th 
questionnaire in which teachers were asked their present perceptions of ^ 
their .pupils. Teachers in high-performing schools >ssw children in their 
classes as more intelligent, better behaved, more pleasant to teach, 4nd 
having more concerned parents than did teachers in low-performing schools. 

Also derived from the teacher questionnaire were teachers expec- 
tations for various kinds of -support within, the school. These data are 
provided in Table 8. 
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Table 7 

Mean Perception of Pupils Expressed by 
Teachers in High- and Low-Performing Schools 



Teacher's Perceptions 
o'f Pupils 


High-Performing 
Schools N=48 


Low-Performing 
Schools N=51 


Intelligence 


3,40 


2.75 


Behavior 


3.27 


2.86 


Pleasant to teach 


3.94 


3.45 


'Concern of parents - 


4.02 


3.09 



• Table 8 •* % 

Mean Expectation for Support Expressed by 
Teachers in High- and Low-Performing Schools 



Expectation for Support in 


High- 
Performing 
School s 
N=48. ■ 


Low- 
Performing 
Schools 
N=51 


Getting needed instructional material 


3.75 


3.49 


Dealing y7ith a behavior problem 


3.13 


3.33 








Developing my own teaching skill 


2.66 


■ 2.75 


Dealing with specific learning problems 


3.19 


3.26 



As can be seen, responses of teachers in high-perf ormipg and low- 
performing scl^ols were very similar. It appears from t]ie»e data that, 
difference in ^tdministrative support as perceived by teachers is , not a 
critical factor. . , * 
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Principal Interview 



The principal in each school was interviewed throagh the use of 
a relatively unstructured procedure. The interviewer then attempted, to 
score a number of responses. One set of items related to the principal's 
e\^luation of rapport and competency of various staff members. These 
data are reported in Table 9, As can be seen, principals in high- 
performing schools provided f<;pnsistently higher ratings than did ^ 
principals in low-performingWhools. All average ratings fell above 
the midpoint of the Likert sc>te, indicating that, on the average, 
principals gave favorable ratings. 

Table 9 , 

Principal Ratings of Personnel 
as Ascertained from Interviews 



Principal Ratings of 


Hiih- 
Performing 
ScAools 


Low- 
Performing 
Schools 


— \ /- 

Teacher rapport 


4.50 


4.14 


JCeacher competency 

i 


4,57 


3.86 


Reading teacher competency 


4.43 


3.57 


Librarian competency 


4,60 


4.29 ■ 



Principals were also asked about their own rapport with various 
groups with whom they deal. These data are summarized in Table 10, 
Principals in low-performing schools were rated from the interview as 
higher in rapport with the school board, but lower in rapport with > 
teachers, parents, afid children. Especially noticeably was the differ- 
ence of the two groups in ratings for rapport with children. Howeye;r, 
because the number of principals in each group was only seven, con- ' 
elusions must be tentative* ^ 

Other data, available from the Principal Interview form, related 
to differences in program organization, structure of the reading program, 
and the like, I^i general, these variables did not differentiate high- 
and low-performing schools; therefore, they are not reported here. 
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Table 10 



Principal Ratings of Rapport as 
Ascertained from Interviews 



Rapport with 



High- 
Performing 
Schools 



Low- 
Performing 
Schools 



/ 



School board 



3.83 



4.33 



Teachers 



4.57 



4.00 



Parents 



4.43 



4.14 



Students 



4.43 



3.43 



Observation of Reading 



The Observation of Reading form was designed to be used when 
the teacher was engaged in direct' reading instruction. When arrange- 
ment's for school visits were made, a desire was expressed to 
observe some reading classes, if possible, but- also to see other 
activities. Since o'bservers were in classes most of the school day, 
and reading was normally scheduled in the morning, reading classes 
were of^en not available. 'In Table 11 the mean ratings are presented 
for activities carried on in reading classes in high- and low-performi 
schools. As can be seen, relatively few reading classes* were observed 
in grades four to six. 



In grades one to three, more total acti\aty seemed to occur in 
the high-performing schools. On nine of the eleven activities more 
emphasis was rated for high-performing schools than in low-performing 
schools. The two areas in which higher means .were recorded for low- 
performing schools were for reading* orally and for management instruc- 
tions. The greatest difference between high- and low-performing school 
was in the relatively large amount of silent reading going on in high- 
performing schools. Thus, in classes in high-performing schoots, ^ 
children were observed more often reading silently; in low-pfer forming 
schools more children were observed reading out loud. Although the 
number of. classrooms was very small, this relationship between silent 
and oral reading extended to grades four to six. 





Table 11 

Mean Ratings of Activities Carrie<d on in Reading 
Classes in High- and Low-Performing Schools 





Grades 1 


-3 




Grades 4-6 


^c tiviCy 


High- 
Perf orming 
School s 
N=34 


Low - 
Performing 
Schools' 
N=38 


High- 
Performing 
Schools 
N=23 


Low - 
Performing 
Schools 
N=25 


\^Liuipr Client xun 


3.22 


(18) 


2 


.45 


(22) 


" 3.50 


(8) 


3.33 (3) 


X 1. U il Llll^ J. Ct I. J. 11 CtllU WULvl 

Recognition 


3.32 


(19) 


2 


.92 


(27) 


3.50 


<6) 


4.50 (2) 


Language Structure 


2.69 


(16) 


2 


.39 


(23) 


3.17 


"(6) 


-2-.75 (4) 


Reading Silently 


3.75 


(12) 


2. 


.47 


(15) 


3.20 


(5)- 


2.50 (2) 


Reading^Orally 


3.40 


(15) 


3 


.68 


(22) 


2.50 


(4) 


4'. 50 (2) 


Spelling \ 


2.92 


(12) 


2 


.23 


(•17) 


2.40 


(5) 


5.00 (1) 


Writing 


2.25 


(4) 


1 


.00 


(6) 


1.00 


(1) 


" ' (0) 


Copying 


2. -40 


(5) 


1 


.63 


(8) 


2.00 


(3) 


3.50 (2) 


Listening Instructions 


2.18 


(11.) 


1 


.00 


(7') 


2.60 


(5) 


3.90 (2) 


Non-reading Instructions" 


1.75 


(4) 


1 


.55 


(11) 


1 .00 


(2) 


1.50 (2) 


Management Instructions 


1.69 


(13) 


2 


.25 


(27) 


1.60 


(5) 


1.50 (2) 



for that activity under each category. 



Also examined in the Observation of Reading Form were the * 
behaviors of children who were not in reading groups* These data are 
reported? in Table 12. ^Again, it should be noted that the number of 
classes involved in reading was small, so the differences seen are not 
necessarily statistically significant. 
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Table 12 

Means of Ratings of Activities of 
Non-Reading Groups During Reading Period 



1 

Activity 
r '- 


Grades 1-3 


Grades 4-6 


High- 
Perf orming 
Schools 


Low - 
Performing 
Schools 


' High - 
Performing 
Schools 


Low 
Performing 
Schools 


Reading'^lently 


3.00 


3.30 


3.27 - 


2.00 


Writing , 


2.56 


2.33 


2.67 


2.50 


Copying 


2 .92 


2.25 


2.86 


3.50 


Non-reading instruction 


2.77 


2.84 


. 3.25 


4.50 


Play 


1.75 


. 2.10 


1.00 


1.50 


Social Interaction 


2.11 


2.18 


2.00 


1.60 



Reading Teacher Interview 

An interview was held with a reading teacher in each of the 
schools visited. The reading teacher was asked to evaluate a number 
of aspects of the classroom reading program. These data are reported 
in Table 13. It should be noted that the averages in the table are 
based on the responses of the seven reading teachers in the high-* ^ 
performing schools and the seven in the low-performing schools. Although 
the small sample size did not permit statistic*al analysis, a definite 
trend can still be seen since higher means were recorded for the high-, 
performing schools in nine of the ten contrasts. Only on item six, 
which concerns the degree to \mich reading material matches the back- 
ground of the child, were low-performing schools given a higher mean. 

Also, of interest were the absolute scores obtained from the 
five-point Likert scale. In general, absolute set -s were higher than 
the scores assigned through direct classroom obsei ation. It is not 
surprising that reading teachers should have ge: -rally seen the program 
in their school in a favorable light. Nevertheless, ^especially in the 
low-performing schools, most ratings fell between 3 and 3.5, suggesting 
that reading teachers saw room for improvement^.. 



Table 13 

Mean Responses of Reading Teachers in 
^ High- and Low-Performing Schools 



Aspects of Reading Programs 


Hieh 
N=7 


Low 
N=7 


1 . 


Variety o'f material 


3.67 


'3. 25 


2. 


Appropriateness of use 


4.17 


• 3.13 


3. 


Effort to extend reading 


3.67 . 


3.00 


4. 


Efforts to ask questions' 


3.16 


3.00 , 


5- 


Efforts to give* purpose 


4.20 t 


3.0Q - 


6. 


Reia.tibnship of material to 
childrens' backgrounds 


3.00 


3.50 


7. 


Use of informal diagnosis 


4.33 


3.29 


8. 


Flexibility of grouping 


•3.83 


. 3.38 . ■ 


9. 


Effective use of test data *^ 

/ 

Availability of interesting 
^books 


3.50 


3.38 


.0. 


4.33 


2.88 



Teacher Reinforcemen t Scale 

- — - » # 

THe Teacher Reinforcement Scale offered a record of the frequenc 
and strength of positive and negative reinforcement. Ratings were r ade 
for reinforcement specif ically 'related to instruction and for general 
support. Data are reported in Table 14. As can be seen, teachers in 
grades one to three in high-performing schools provided^more positive 
reinforcement under all conditions, and teachers in low-performing 
schools provided more negative reinforcement. However, in grades 4-6 
teachers ii? low-performing schools provided more reinforcement, both 
positive and negative, than did teachers in high-performing schools. 

In the instructional-specific category, teachers in al.l groups 
were recorded on the average as providing more positive reinforcement 
than negative reinforcement. Under the category of general support, 
however, teachers in high-performing schools h^d 'a higher frequency of 
positive than negative reinforcement, while teachers in low-performing 
schools showed the reverse pattern. Only two mean ratings reached the 
midpoint of the five-point scale. This indicates that, in general, 
the observers did not see a great amount, of reinforcement. 
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Table 14 , ' r 

Mean Scores for Teacher Reinforcement in 
Classes in High- and Low-Perfbrming Schools 



Type of Reinforcement 


Grades 1-3 


Grades 4-6 


High- 
Performing 
School *^ 


Low- 
Performing 
Schools 


High- 
Performing 
Schools 


Low- 
iPerformi ng 
Schools 


Instructional -specific 










Positive 


3/08 


, 2.58 


2.27' 


• 2.42 


Frequency 










Negative 


1.47 . 


1.87 


.1.68 


■ 1.92 


Positive 


2*97 


2.79 


2.31 


• 3.00 


Potency 










Negative 


. 1.81 


2.37 


1.86 


2.58 • 


General Support 


• 








Positive 


2.56 


• 

1.82 


2,09 


2.16 


Frequency 










Negative 


'1.53 


2.58 


1.52 


2.28 


Positive 


2.76 


2.47 


2.33 


'2.96 


Potency 










Negative 


■ 1.90 


2.66 


1.^6 


2.88 



Characteristics of Open Education 

• The form, Some Characteristics of Open Education, is a shortened 
version of a scale developed by Walberg and Thomas^^ to measure the 
degree of classroom openness. Since this form was not used in'^relfm- 
inary field work, no data, were available, to form a basi»^for revision*. 
Observers reported that some items^were not really suitable for obser- 
vation and also indicated that pola^ definitions would have been useful 

However, in spite of the^se problems, a definite trend was seen 
in scores for high-performing and low-performing schools* Of the 4-8— ^ 
comparisons in grades ^-3, 15 were ,in favor of high-performing schools, 
as can be seen in Tab^le 15. This Crend was, not strong in grades 4-6, 
in which means for highrperforming .schools were greater in 11 of 18 
comparisons* For almost all items of open education characteristics, 
ratings were below the midpoint of the five-point scale* Thus, one 
might assume that the gene^^al sample of schools employed in this study 
was relatively traditional, and this assumption was verified by the 
opinions of the observers. 
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Table 15 

Comparison of Mean Scores of' Characteristics' of Open 
Education in High- and LoxN?-Performing Schools 



Teacher Characteristics 
(Abbreviated) 



^£1 
1^ 



Higl 
Performing 
Schools 



Low- ^ 
Performing 
Schools 



Grades *4-6 



High- 
Performing 
Schools 



Low - 
Performing 
School-B 



L. Time to individuals, 
not group 

2. Teacher uses child 

interaction 

3. Teacher encourages 

independence 

4. Individual -child diag- 

nosis 

5. Toac^ier encouraj:*'^ 

i\.ntisy 

6. , Diverse instructional 

materials 

7. >hceri ds acco5;sibUi 

8. Chi Idre-n move la^^ ly 

9. Children Ui>o other nifas 
10. Many diverse activities 

^11. Much individual work » 
'12, Individual evaluation 

13. Individual instruction 

t 

14. Teacher respect for 

child 

15. Children freely 

express feel ing 

16. Teacher promotes trust 

17. Teacher keeps le a'ning 

18. Teacher is secure 



2.51 

2. 16 

1.48 

2.30 

1.67 

2'.53 
2.9^. 
2.41 . 

i.ir 

2.21, 
1.8L 
2.53 
2.»62 

2.63 

2.67 " 

3.04 

2.64- 

2.09 



2.45 

1.75 

1.71 

2.37 

1.16 

1.97 
2.57 
2.13 
1.12 
2.02 
1.63 
2.05 
2.07 

1.97 

2.29 

3.54 . 

2.34 

1.92 



2.57 

2.06 

1.47 

2.00 

1.38 

1.75 
2.3-5 
2.13 
1.50 
1.86 
1.71 " 
1.79 
2; 08 

2.Q0 

2.29 

2.26 
2.22 
I .72 



2.08 
.2 .'00 
■ 1.59 
1.82^ 

1. L6^ 

1.61 

2:17 

1.75 

1 .'24 

1.48' 

1.39 

I'.OO 

2.10 

. 2:33 . 

2. *48 

2.57 
2.26 
2.00 
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Correlations af items of the Characteristics of Open Education 
form were examined, and five items were selected which seemed especially 
observable in classroom settings. Intercorrelations of these items in 
grades 1-3 in high- and low-performing schools are presented in Table 16. 
As can be seen, these items generally were highly correlated, especially 
for high-performing schools. These intercorrelations suggest that it 
may be possible to tap a relatively broad characteristic of openness 
that differentiates classrooms. 



Tajble 16 



Correlations of -^elected Items Related 
to Open Education ^n Grades 1-3 



1. Time to individuals, 

not group 

2. Diverse materials, 

not class sets 

3. Materials readily 

accessible to children 

4. Children move' freely 

without asking per- 
mission 



1 



.18 
.38^^ 
. 53^*^* 



2 

. 80** 



.04 



3 

.76^^* 
.85** 



4 

.81** 
.67** 



5 

.87*** 
. 75** 



.40** 



.73^^* .73** 



. 84**. 



5. Many Activities going 
or. at same time 

**signif icant at .01 
* significant at .05 



.46** 



.33* 



.49^f* 



.49** 



(Classes in high-performing schools are 
given above the diagonal; classes in 
low-per f orming^schools are given below 
the diagonal . ) 



Discrimination Between High- and - 
Low-Performing Schools 

It is desirable to know whether there are clear and reliable 
differences between schools designated as high and low in performance. 
In order to assess the consistency of ranking, Mann-Whitney U's were 
computed on the rank order of high- and low--performing schools on 
certain variables. For selected items on the Open Education .scale. 
Table 17, there was no significant relationship between achievement 
category a'nd rank on this ccale ; high-performing schools were slightly 
Hiore likely to be rated high on this scale. On the General Classroom 
Observation scale. Table 18, however, there was a consistent relation- 
ship with achievement category (U = 3.45), significant at p^.Ol.- 
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High achieving schools generally received higher ranks on this measure, 
although a few reversals were noted. On selected items from the scale 
used for observation of reading, Table '19,' there was also a significant 
relationship (U = 2.81, p<.01) between score on the scale and perform^ 
ance category of the school, with high-peif orming schools generally 
having received higher scores. 



Table 17 

Discrimination of Rank-orders of High- and Low- 
Performing Schools on Selected Items on Open Education 

Performance 
Category 

High 
High 
Low 
High 
Low 
Low 
- High 

HigTr 
Low 
High 
High 
Low 
Low 



Score Rank 

15.1 1 
13.4 2 

11.8 3 

10.9 ' 4 

10.2 . 5 
10. 1 6 
10.0 7 

9.2 8 
9. 1 9 
8,7 10 
7.7 il 
7.6 ♦ 12 

7.3 13 
7.1 -14 

U = 1.66 N.S. 



Table 18 
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Discrimination of Rank-orders of High- and Low-Performing 
Schools on Selected Items on General Classroom Observation 



Performance 

Score Rank Category 

35.30 H 
33.18 2 



High 
High 



^ 33.14 .3 High 

31.30 4 ,Low 

31.20 5 Low 

29.56 ,6 High 

29.25 7 • High 

27.48 8 High 

27.81 9 Low 

27.30 10 Low 

27.23 11 Low 

27.10 12 High 

25.86 13 Low 

24.68 \ 14 Low 
U = 3.45 significant at .01 
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Table 19 

Dis^irimination of Rank-orders of High- and Low-Performing 
Schools on Selected Itdms on Observation of Reading^ 



bcore 

15,66 
13.00 
13.00 
11.66 
11.60 
11.04 
11.00 
10.75 
9.33 
8.95 
8.76 



Rank 

1 

2 
3 
4 
5 
6 
7 
8 
9 

10 

11 



Performance 
Category 

High 
High 
High 
' . High 
High 
Low 
High 
Low 
Low 
Low 
Low 



U = 2.81 significant at .01 ^ 

^-Threo schools are excluded from this analysis 
because there were too few occasions in which 
explicit reading instruction was observeci. 



Observer Agreemenl 

7 

A number of options presented Lht-mselves as procedures which 
might have been followed to clieck observer agreement. Separate reli^- 
abilitios could have been computed for G'H h form, or vafious techniques 
of computing observer agreement could have been followed. A simple 
procedure which seemed adequate tt thib r.tage was to obtain root mean 
.squares usim; the lormula: 



root mean square = 



" r 

where n = number of variables and d = difference in 
ratings between two observers on a variable, , 

Using this procedure, the possible range o£ scores is zero tp four. A 
score of zero would indicate perfect agreement on all seventy variables 
scores on all observational instruments. A score of foui^ would indi- 
cate complete disagreement. Table 20 shows the root*mean squares 
obtained for each pair of observers. In general, substantial agreement 
was found. However, observer one showed considerable disagreement with 
observers fqur and six. 
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Table 20 





Root Mean 


Square Differences 


Between 


Observers 




Observer 


2 3.4 


5 


6 7 




1 ■ 


i.02 1.81 


1.06 


1.60 




■' 2 


.84 


.78 


1.20 .97 




3 




1.25 






4 




1.00 


1.13 1.13 




5 


•* 




.51 




• 6 






.83 



It should be noted that the root mean square procedure would 
show disagreement if observers had different response tendencies, 
which standard reliability test's would not pick up. In other words, 
one observer /might score very strictly and one quite leniently, but 
they might a^ee very i^ll in the profile of their data. The two 
observers would be crecflted with 'high reliability. Under a root 
square method, however, the absolute difference between their ratings 
was used which provided a more stringent criterion. 

At this stage of instrument development, the degree of relation- 
ship found between raters was seen as encouraging, but this should 
remain an area of future study. 



Discussion ' X 

3? 

In general, the procedures followed in this study seemed to 
produce meaningful differentiations between schools identified as either 
high or low in academic performance. Data were generally in harmony with 
the findings of other studies. 

There are several steps that might be taken next: s • 

1. Select the variables that seem to be related to the clearest 
differences between schools. 

Attempt to clarify further the behaviors that are being 
rated and the criteria for rating each of these variables. 

3. Consider whether the variables identified by these proce- 
dures could be meaningfully divided into sub-parts to^be more specifi- 
cally studied. 

4. Try out revised materials on a broader geographic basis and 
with schools that are demographic a 1 ly more diverse. ' 
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5. Experiment with these materials in 'an in-service and/or 
pre-service context, 

6. Work systematically with a group of teachers to see if 
teachers can learn to vary selected behaviors, and study the effects 
of such variations. 

^ Summary of Most Significant Findings 

Listed be-low, are seventeen conclusions that emerged from the 
data. Most of the conclusions "focus on differences between the high- 
and low-performing schools, but a'few involve no* difference findings. • 
Much could be written on the impl ications' of each'of these conclusions, 
but on another level the conclusions speak for therasfelves^. They should 
" be taken as tentative *and subject to further confirmation. 

1. Teachers in high-performing schools raade^esS overt effort 
to maintain c.lass control than those in ^ow-perforraing schools, had 
less rigid student behavior, but were" more* effect ive in maintaining 
the level of control they appeared to want. 

2. Teachers- in high-performing schools vere rated as warmer, 
more responsive, and showing more emphasis on cognitive development in 
classes tliat did not involve direct reading instruction as well as in 
reading classes. ^ 

3. Teachers in high-performing schools expected more children 
to graduate from high school; to go to college,, to become good readers, 
and to become good citizens. ' 

4. Teachers in high-performing schools saw their children as 
more intelligent, better behaved, more pleasant to teach, and their 
parents as more concerned. 

5. Teachers in .high- and low-performing schools did not 
perceive different amounts of help available to them in handling problems. 

6. More total 'activity takes place in* reading classes in high- 
^ performing schools than in low-performing ^hools. < 

7. Children in 'reading classes in high-performing schools 
engaged in more silent reading while children in low- performing schools 

^engaged in more oral readifig. 

8. Re*'ading teachers' evaluations of classroom reading programs 
were on the average more favorable in high-performing schools. Reading 
teachers rated teachers in high-performing schools more favorably than 
they did teachers in Tow-performing schools in using appropriate material, 
extending .reading into other areas, 'asicing* chi Idren to read v;ith a pur- 
'pos'e, and usiog^informal* diagnosis/ • 

50 . • 




. _ -51- 

9, In grades one to three, teachers in high-performing schools 
gave more positive, and Less negative reinforcement than did teachers 
*in low-performing schools. ^ J 

IQ. In grades four' to six,, teachers in low- per forming school's 
gave more reinforcement. .In general,, however , teachers 'in high- 
performing schools gave positive reTnforcement more than negative 
reinforcement. 

'11* On selected items related to open education, high-performing 
schools appeared more open than low-performingv Schools. 

12. On several measures, differences between high- and low- 
performing schools seemed more pronounced in grades one to three than 
in grades four to six. ^ ^ . 

13. Of three instruments tested, two significantly di'fferen- . . 
tiated high- and low-pei^rming schools arranged* on a rank-order basis. 
Thus, not only did individual items differentiate between schools, but 
tatal rankings on the two instruments distinguished successfully between 
the two groups of schools. * ^ . , 

14. .Principals in high-performing schools generally saw their 
personnel, as more 'competent than did principals in* low-performing, 
schools. I ' . • 

15* Principals in high-performing schools saw themselves as 
having better rapport with teachers, parents, and put)ils than^did'* 
principals in low-performing schools. * Principals in low-performing 
schools reported better rapport with their sc>iool boards, however/ 

— 

16. Items on physical space and facilities gene'ralTy did tiot 
differentiate between high- and low-performing* schools. 

17. In general, i^easonably cl6se relationships were found 
between the ratings of different observers. 



■J 
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